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ABSTRACT 



The effectiveness of an evaluation model for 



interactive television courses was studied. Inquiry was conducted 
with distance education graduate students using a structured, 
open-ended questioning format to obtain their authentic voices about 
a traditional model: Small Group Instructional Diagnosis (SGID) , a 
facilitator~based model how to open the lines of communication 
between faculty and students. Subjects responded to three questions: 
what helped their learning in the program?; what hindered their 
learning?; and what changes would be suggested? At the conclusion of 
two courses in Telecommunications and Technology for Teachers, 
participants were asked to evaluate this SGID process using a version 
of SGID modified for delivery as a computer-mediated communication. 
Analysis of students' on-line comments and administrative actions 
indicated that SGID is effective for these students. Participants had 
a high level of comfort in sending messages to each other, the 
instructor, and a discussion list. Student interaction between the 
process and facilitator was cited as an important aspect in the 
helpfulness of the process. Items on hindering student interactions 
with the process were identified by more remote students than by 
those on campus. On-site students were more apt to indicate no 
interact ions hindered the process and that no changes were needed. 
Administrators demonstrated change activity. These results are 
consistent with the data on SGID and have potential for those 
evaluating graduate courses delivered by two-way audio and video. 
(Contains 27 references.) (Author/MAS) 
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Abstract 



The effectiveness of an evaluation model for interactive television courses was studied. Inquiry was 
conducted with distance education graduate students to obtain their authentic voices about a traditional 
model: Small Group Instructional Diagnosis (SGID), a facilitator-based model. Analysis of students' on-line 
comments and administrative actions indicated that SGID is effective for these students. Student interaction 
between the process and facilitator were cited as important in the helpfulness of the process. Items on 
hindering student interactions with the process were identified by more remote site students than by those 
on campus. On-site students were more apt to indicate no interactions hindered the process and that no 
changes were needed. Administrators demonstrated change activity. These results are consistent with the data 
on SGID and have potential for those evaluating graduate courses delivered by two-way audio and video. 



Applying an Interactive Evaluation Model to Interactive Television 
Introductior 

The use and the effectiveness of student evaluation concerning course offerings and instructor 
effectiveness have been documented in both distance education and traditional classroom settings. 
Numerous studies of distance education incorporate student evaluations as one or more of the tools used to 
assess course effectiveness (Dillon, Gunawardena, & Parker, 1992; Schlosser & Anderson, 1994; Wagner, 
1993). Maitino's review of current research on student evaluation of college instruction in traditional 
classrooms (1992) supports the reliability and relevancy of such data. 

In 1979, Clark and Bekey refined a University of Massachusetts clinical model that provides mid- 
semester evaluative feedback to the instructor based on oral information given by small student groups to an 
external, trained facilitator who spends one class period alone with the students. During the period, after 
separating the students into small groups, the facilitator directs all the students in each group to provide 
answers to two questions about everything that helps and hinders their learning in the course. A third 
question is also posed that asks for their input on any needed changes. After a recorder from each group 
provides all the responses, the facilitator leads the class in clarifying the responses to assure mutual 
agreement on meanings. The class then votes on each response, indicating the degree to which each 
student supports each item. After the class session, the facilitator meets with the instructor, providing the 
ranked results of the student responses, interpretative comments, and suggestions for improving instruction. 

This process, the Small Group Instructional Diagnosis (SGED), that has appeared in the literature 
on teaching effectiveness as one type of evaluation technique, has been shown to open the lines of 
communication between faculty and students and (Seldin, 1988). 

Recently there has been an increased interest in conducting research that encourages the application 
of research methods that produce results leading to immediate change and supporting improved practice in 
teaching. While there is debate over the distinction between practical classroom inquiry and "teacher as 
formal educational researcher" (Richardson, 1994, p. 7), a distinguishing feature of the latter is applicability 
of results to a broader community. Such qualitative studies have the potential of providing researchers with 
opportunities for describing natural habitats as they are with minimal disturbance; providing insights to 
existing themes and addressing immediate needs. 

No one source of data is sufficient for evaluating the efficacy of an instructional situation (Seldin, 
1988). While a concern facing those engaged in qualitative studies is structuring and validating the 
resulting data, triangulation is an approach that lends support to the analysis. Through comparisons of 
interview data with observational data and anecdotal reports, and by investigating participants' comments 
and behaviors, patterns can often be seen to emerge that support hypotheses, or lead to revised insights 
(Patton, 1987). This study, while based primarily on data gathered from student reports on their 
perceptions of the effectiveness of a formalized evaluation process, also provides data from actions of the 
stakeholders: program administrators, faculty members and Department of Education teachers. 

A structured, open-ended questioning format familiar to the distance education students was selected 
to provide a vehicle in which their authentic, diverse voices could be heard as they provided their 
perceptions of learning in two distance education courses. 
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In the body of work on distance education, the emphasis on evaluating the impact of distance education 
courses can be categorized as: 

• Comprehensive: The overall approach and design of evaluatior of distance education (Thorpe 1988" 
Zvacek, 3994). 

• Administrative: The uses of distance education (Dillon, Gunawardena, & Parker, 1992) and its costs 
(Willey, 1991; Zvacek, 1994). 

• Instructional Design: The study of student learning in terms of suitability of courses (Willey, 1991) 
and student perceptions of interaction (Fulford & Zhang, 1994). 

• Learner Achievement and Attitude: The factors contributing to student success and positive attitudes in 
distance education situations (Dillon, Gunawardena, & Parker, 1992; Lauzon, 1992; Schlosser & 
Anderson 1994; Verduin & Clark, 1992; Walker & Hackman 1992; Willis, 1991). 

• Technical: The effectiveness of distance education courses in terms of delivery systems (Egan, Welch, 
Page & Sebastian, 1992; Schlosser & Anderson, 1994). 

Among the emerging trends in distance education (Birenbaum, Click, & Kinsler, 1994), is the 
continued emphasis on examining the effects of distance education on the learner. This study used such a 
focus — with the focus on learners who were involved in a new cooperative distance education university 
and school v. .partment partnership. By employing a broadly structured, open-ended format for obtaining data 
from the learners, this study allowed student input in any of the areas for evaluating distance education: 
comprehensive, administrative, instructional design, learner achievement and attitude, and technical. Given 
the structure of the inquiry, one anticipated outcome was that the resulting data would have an emphasis on 
learner related issues, a result which was supported. Similar to studies by others in the field of distance 
education research (Holmberg & Bakshi, 1992; Walker & Hackman, 1992; Kember, Lai, Murphy, Siaw, & 
Yuen, 1992), this study relied on a survey for gathering information from the participants. 

At the University of Hawaii, Small Group Instructional Diagnosis has been employed for the past 
five years by its Center for Teaching Excellence, a unit within the Office for Faculty Development and 
Support. At the mid point of the semester in traditional, intact classes throughout the campus, trained 
faculty and teaching assistants from the Center assist faculty in assessing the effectiveness of their courses 
by obtaining student perceptions about their learning. In addition, since 1969, when the University of 
Hawaii offered the first college credit course delivered by satellite, the university has been developing its 
interactive television facilities. The Hawaii Interactive Television System (HITS) currently delivers a wide 
range of courses at the undergraduate and graduate levels using a technologically advanced delivery system 
that incorporates two-way audio and video at its origination site on the island of Oahu and at its receive 
sites on the neighbor islands of Kauai, Hawaii, and Maui. To date, evaluation of HITS courses and 
programs has been accomplished through written responses provided by both students and HITS instructors 
in end-of-semester questionnaires. None have incorporated the unique interactive feature offered by HITS 
into the evaluation process. 

With the SGID model and HITS at the University of Hawaii, the resources were available for 
undertaking a preliminary investigation of the effectiveness of applying an interactive classroom evaluation 
procedure, SGID, to introductory courses in a cluster of five Department of Educational Technology 
(ETEC) graduate courses, entitled Telecommunications and Technology for Teachers (T^), delivered via 
interactive television. Although the SGID model has been used in traditional class settings, it had not been 
used in a distance education class at the University of Hawaii and no instances of its use in other distance 
education classes were discovered. 

The department faculty determined to apply the SGID at mid-semester to the T 3 program to obtain 
timely feedback on the program and to assess the effectiveness of this evaluation technique in a distance 
education setting. 

Questions under investigation were: 

Question 1: Would a modification of the SGID, delivered as a computer-mediated communication, 
provide data of sufficient quality and quantity for use in a study of SGID as a distance education evaluation 
process? 

Question 2: Would a cohort of graduate students enrolled in a cluster of distance education courses 
differ in their perceptions of a course evaluation process, the Small Group Interactive Diagnosis? Would 
those perceptions be influenced by receive site location; by effects of two-way audio and video during group 
reports to the entire class; by the effects of video graphics used for ranking results of small groups reports; 
and by the large group sizes that some sites may have during the SGID? 

Question 3: Would the results of the process be available for program administrators and 
instructors in a timely manner, allowing for program and course revisions? 
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Methods and Procedures 



Subisas 

The population for this study consisted of a cohort of fifty Department of Education teachers and 
specialists enrolled in the first two of five graduate level, courses that comprised the Telecommunications 
and Technology for Teachers (T 3 ) Program. The T 3 courses were scheduled two nights per week for the fall 
and spring semesters of 1993-1994, with the final course scheduled in the first 1994 summer session. 
While these courses were part of the graduate program offered within the Department of Educational 
Technology at the University of Hawaii, they did not of themselves comprise an official program of study. 
The Department of Education acknowledged the completion of the five courses with a T 3 Certificate of 
Completion, which led to the informal designation of the courses as the T 3 Program. 

Students were located at three remote sites: the islands of Maui, Kauai, and Hawaii. T 3 students 
were also present at the main campus transmission site on the island of Oahu. Ninety percent of the 
cohort, 45 students: 21 from the on-campus site and 24 from the remote sites, participated in the Small 
Group Instructional Diagnosis during the ninth week of the 15 meeting cycle. Following the SGID 
protocol, a trained, external facilitator met with the students for an entire class period without the presence 
of the instructors. The students actively analyzed the effectiveness of the first two courses in the T 3 
Program by responding to three questions about what helped their learning in the program, what hindered 
their learning in the program, and what changes they would suggest for the program. 

At the conclusion of the two courses, the participants were asked to evaluate this SGID process 
using a version of SGID modified for delivery as a computer-mediated communication. 

Eighty-seven percent of the 45 mid-semester participants participated in this evaluation: 19 from 
the on-campus site and 20 from the remote campuses. These 39 students formed the sample for this study. 
Characteristics that the sample have in common with all distant education populations are their adult status 
and the predominance of females, 82%. Their wide ranging ethnic backgrounds, while typical for the state 
of Hawaii, are not typical of other distance education students. This study is limited by its intact 
population. 

Meas ur e 

Computer Mediated Communication SGID: The quantity and quality of responses to the on-line 
delivered survey questions were assessed by amount and quality of information conveyed. A response rate 
indicative of 75% was established as a way to assess the quantity of the returns (Seldin, 1988). Response 
quality was assessed by the degree to which the respondents provided usable information for each of the 
three questions. 

Perceptions of the SGID: To assess the extent to which the subjects differed in their perceptions 
of SGID, a modified version of the SGID process itself was used. As a qualitative measure, it has no 
reliability coefficients; its construct validity is attested to by use by others since 1979 (Clark & Bekey, 
1979; Seldin, 1988; Theall & Franklin, 1990). The overall impact of the SGID was assessed through a 
comparison of the number of responses for each of the three questions in relation to positive responses. 

The three questions designed to elicit open-ended responses about the positive and negative aspects 
of the evaluation process, as well as to elicit suggestions for changes in SGID, were sent to the on-line T 3 
discussion list The request and survey were transmitted to the T 3 students on the discussion list that had 
been established and extensively used throughout both courses. To participate in the study, the students had 
to have participated in the SGID administered at mid- semester. Responses were sent to the primary 
researcher. Attesting to their interest in evaluating the process, in addition to responding to the researcher, 
over 40% of the respondents elected to send a copy of their responses to the T 3 discussion list, following 
the T 3 program protocol of sharing information of general interest with all participants. 

Patterns of concepts and frequency of words and phrases related to the variables of site, effect of 
delivery mode/graphic display, and group size were set as the way to interpret responses related to those 
variables. 

Program Impact: To study the impact of the results of the SGID on the T 3 Program, the log of 
administrative actions occurring after reception of the SGID results was analyzed by types of actions. 
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Data Analysis 

Question 1: To assess the viability of using the on-line generated responses, the response rate was 
computed for both inter group and total group and compared to the minimal desired rate of 75% (Seldin, 
1988). 

Question 2: To assess differences in the perceptions of the distance education students at both the 
on-campus and remote sites, student responses relevant to the three SGDD questions were carefully reviewed, 
separated into discrete phrases, coded, and entered into a database and analyzed. 

During the preliminary analysis of the phrases, a conceptual framework emerged that appeared 
related to the three interactions identified by Moore (1989) regarding distance education settings. His 
"learner-content interaction" emerged as "learner-format interaction" for this analysis. Participants' 
responses, such as, "The three questions were to the point," were placed in this category. Moore's "learner- 
instructor interaction" emerged as "learner-facilitator interaction" with the facilitator referring to the person 
who conducted the SGID. Statements, such as, "having a patient and objective facilitator," were identified 
as learner-facilitator interactions. No changes in the semantics for Moore's "learner-learner interaction" 
category were deemed necessary> Statements, such as, "being able to see the number of students that agreed 
with my feelings," were identified as learner-learner interactions. 

The plausibility of this coding was tested with an expert panel, consisting of the researchers and an 
external evaluator. The researchers provided deep understandings from their immersion in the setting; the 
independent expert provided an unbiased view and knowledge as an evaluation specialist Response 
categorizations, made independently by the expert panelists for all items, revealed a high degree of inter-rater 
agreement with the fit of Moore's modified typology. Those items not agreed upon were discussed, 
resulting in mutual agreement on the categorization of all items. 

Further analysis within the typology was carried out with like words and phrases within the 
database. For example, "continue" and "don't change" were considered synonymous. 

Question 3: To analyze program activity generated by the results of the SGID, related occurrences 
were analyzed by investigating the data arrayed on a timeline. 



Results 

Question 1 : The results revealed a response rate of 87% for all sites, with an on-campus site 
response rate of 90% and a remote site response rate of 83%. All response rates exceeded the standard set for 
a minimal 75% return rate, thus, providing sufficient quantity of data for analysis. As indicated in Table 1, 
the data was relevant to the three SGID questions. Of the 213 types of responses, only 6 items were not 
applicable to the three questions: "What helped the evaluation process?" "What made the evaluation process 
difficult?" "What recommendations do you have for changes for the mid-semester Small Group Instructional 
Diagnostic process?" Both the overall high return rate, 87%, and the number of usable items, 207 out of 
213, indicated that the computer mediated communication was an effective process for gathering data for this 
study from participants who were all active in their use of on-line communications. These participants 
were characterized as having a high level of comfort in sending messages and to each other, to their 
instructors, and to the T^ discussion list The response rate may not be rcplicable for new on-line users or 
for users who are not used to corresponding with their instructor in this manner. 

Question 2: To determine any differences between the on-campus and remote site participants, 
within group percentages for responses were calculated and compared. As indicated in Figure 1, few 
differences were reflected by respondents. Approximately 50% of the comments referred to the helpfulness 
of the evaluation process, regardless of the location of the respondent. Less than 26% referred to hindrances, 
or need for modifications, for SGID. An examination of Table 1 reveals that some differences do appear 
within sub categories of the three main categories, with the most marked differences appearing in relation to 
hindering interactions. At the remote sites, 74% of the responses given by the participants, such as, 
'Towards the end of the session, we felt pressed for time," were categorized as being learner-process 
interactions, in comparison to only 50% of this type of comment being given by participants at the on- 
campus site. 

A new type of sub category emerged; one reflecting, "no problems". While actual numbers are 
small for meaningful interpretations of this subcategory, it is interesting to see the patterns that appeared 
within both the hindering interactions and changing interactions categories. At the on-campus site, a total 
of eight comments, such as, "I don't think that there was anything that made the process difficult," in the 
hindrances group and eight, such as, "I would recommend we continue to have mid-semester diagnostic 
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feedback sessions," in the modifications group were made. In both instances, however, only two or less 
were made at a remote site. The remote site also offered 1 8% more comments on ideas for changes related 
to learner-process interactions than did the on-campus group. These comments included, "Perhaps a faster 
way of counting the 'votes' for each of the items. Maybe using the fax machine to facilitate this." "Also, 
perhaps limiting each g. ip to 3-5 answers to each question. I think this would eliminate some answers 
that were repetitive and some that were 'off the wall' and subsequently received few votes." 

After categorization of items by the panel into types of interactions, like items were further 
collapsed to create items of similar concepts, so that "professional manner", "professionally done", 
"excellent job", and , "leadership skills" were categorized as overall skills of the facilitator. These items 
were ranked by sites based on frequency counts within groups. The results of this ranking, displayed in 
Tables 2 through 4, show, that regarding helpful interactions, strong agreement was reflected at all sites 
regarding the excellent overall skills of the SGID facilitator. This item was mentioned more than any other 
for all groups, as well as by the remote site group. For the on-campus group, this item was in a three-way 
tie for first place with "the facilitator's ability to create a supportive atmosphere" and in the learner-learner 
interaction subcategory, with "professionalism of the group". Tables 2 through 4 provide a comparison of 
items that respondents cited most frequently. Unanimity was reached across groups on hindering 
interactions within the learner-process interaction subcategory with "the amount of time for the process" 
being the most frequently mentioned item being perceived as a problem and, similarly, in the change 
interactions category' with "conducting earlier in the semester". 

An analysis of the number of times that specific references to any constraints imposed on the 
evaluation process by the effects of two-way audio and video on the group reports and of the video graphics 
used to display the group reports for voting, showed that these items received specific mention only once 
each at the remote sites and no mention at the on-campus site. The remote site comments were: "It was 
difficult not being in the sam> room because comments were made that [were] not stated openly." "TV 
made it difficult to keep track of all ihe comments since all of the writing had to occur on the elmo [visual 
presenter]. Still this is a limitation that cannot be helped." One positive comment was received from a 
rem' re site in regard to the vistialization of the group's ideas: "I also think it helped that [the facilitator] 
wrote down all the ideas and paraphrased to ensure agreement." 

There were more item;V related to the large group sizes with five comments, 6.8%, from the on- 
campus site regarding learner-le \rner interaction problems: "I did notice some people did not volunteer 
much for discussion. It was their choice though." The remote sites offered three negative comments, 5.3%, 
in this regard. A typical comment was, "The group was a little too large." At the same time, both the on- 
campus and remote site participai-ts made ten comments, 13.5% and 17.5% respectively, regarding the 
helpfulness of the learner-learner interactions: "I liked how it was directed and that we were broken up into 
groups so our 'voices' could be heard collectively". "Group memory kept the group focused on the tasks." 
These results indicate, that for these participants, the issue of technical issues and group size were not 
predominant concerns. 

Question 3: To analyze the data in relation to question 3, to see if the results from the Small 
Group Instructional Diagnosis provided results in a timely manner to allow for program and course 
revisions, the activities that the process engendered were plotted on a timeline. Figure 2 shows the increase 
in activity immediately prior to the administration of the SGID, and the subsequent activity following the 
process. Of particular note is the appearance of the Task Force on the Technology Specialist. This Task 
Force, charged with the examination of the actual role of the proposed new Department of Education (DOE) 
specialization, had been discussed at prior meetings, but the SGID provided the impetus to bring 
stakeholders from both the ETEC Department and the DOE together to identify the skills needed by 
someone in this proposed position. The ETEC Department also used the results of the SGID for long- 
range and short-range planning. Based on SGID data on course scheduling, courses for the following 
semester were scheduled in tandem rather than concurrently. For the long term, the ETEC Department began 
re-examining the courses selected for the program in light of the new scheduling, with plans for a seminar 
session to be interwoven throughout the courses. 



Discussion 

In this study, a model for evaluating two distance education courses that formed part of a program 
of study, was investigated. The range of data obtained from the distance education participants supported the 
recommendation from a study of seven distance education courses in Hong Kong (Kcmber et al., 1992) that 
others include more than one course in their evaluation to provide a broader range of input. Although the 



524 

7 



Kember et al. study used a mix of qualitative and quantitative methods, their qualitative data was used more 
extensively in their analysis. 

Time constraints posed limits on the manner in which this computer mediated communication 
version of the SGID, used for gathering data on the actual SGID, was executed. The facilitator role of 
restating and clarifying items which could have been attempted on the class discussion list, was omitted in 
the on-line version because of the time constraints imposed by the approaching end of the semester. 

This study provided a connection to the everyday world of educational practice. The high response 
rate to the on-line SGID, in less than three weeks by the Distance Education students, reflected the appeal of 
this modified version for data gathering, supporting Reinharz's endorsement for using "special techniques 
that are "unobtrusive measures" (1993). His support for having good pre-conditions for a specific method 
was supported by the familiarity and positive attitudes of the participants, toward on-line communication. 
Lauzon (1992) also supports on-line communication as an excellent medium for meaningful interactions 
between learners and facilitators, instructors, and each other. A version of SGID modified for use as a 
computer mediated communication did provide useful information for this qualitative study. 

Analysis of the comments was carried out by the expert panelists, using the students' language to 
tell what was considered important, requiring reliance "on their own intelligence, experience and judgments" 
(Patton, 1987, p. 154). The use of Moore's typology (1989) that emerged from the analysis of the 
authentic voices of the participants, gave structure to the analysis of the items and, thus, guidance for 
replicating this study. 

Identifying a modification of Moore's typology, instead of creating their own, also gave the 
authors a way portraying the students' views in a manner that addressed, to some degree, the potential for 
bias that could arise for the authors in their dual roles as participants throughout the SGID process and as 
authors of the study. The external member of the codifying panel attested to the validity of applying the 
adapted version of Moore's typology. 

The emergence of three categories of the interactions of learner-format ("learner-content" in 
Moore's categorization), learner-facilitator ("learner-instructor" according to Moore) and learner-learner 
supports the distinctions being made in the field of distance education of the importance of identifying 
typologies for interactions. Such schemata lead to improved communication and study within the field 
(Moore, 1989). It is of note that two types of responses did not fit his typology; these responses are the 
items that were cited when participants were asked to name hindrances or changes to interactions. The 
responses of "no hindrances" and "no changes" were noted in separate categories established to accommodate 
this type of response. 

The data from this study suggests use of the Small Group Instructional Diagnosis as an effective 
evaluation model for distance education students enrolled as a cohort in graduate level education courses. 
More than 50% of the response items related to factors that helped the process, regardless of the site; with 
only 21% of the items at the remote sites and 16% of the items at the on-campus site related to items that 
interfered with the process. This finding supports an earlier evaluation (Clark & Bekey, 1979) of the 
process. In Clark & Bekey's evaluation of the effectiveness of SGID for students in a traditional educational 
setting, 80% of the participants stated that SGID was useful when asked to respond in either the affirmative 
or the negative. 

While a little less than 7% of the distance education, campus-based students registered their 
enthusiasm for the SGID in statements on "continuing/not changing the process" and in requests "for using 
the process" in their roles as teachers, no similar remarks were noted from the remote site students. The 
remote site participants made quantifiably more hindering interaction statements than the on-campus 
participants, although both types of participants identified "the amount of time for the process", "time 
limits", and "peer pressure" as their top concerns. The remote site participants expressed a greater range of 
comments, from statements such as "the voting didn't quite express the ideas", "negative feelings", and 
"versions varied by site". This wider range could be indicative of differences that may have occurred from 
aggregating the responses from three different remote sites into the sole category of "remote site" and from 
two of the remote sites having less options for forming varied groups than the on-campus site and the third 
remote site, both of which could accommodate three groups of seven . 

The size of the groups, most of which consisted of seven participants, and as such violated by one 
person the maximum number recommended for groups in the SGID (Clark & Bekey, 1979), may account 
for the concerns expressed about time. The additional person may impact the amount of time available for 
optimal learner to learner interactions. 

Although reports of SGID may omit mention of, or not stress, administration by a trained 
facilitator (Theall & Franklin, 1990; Seldin, 1988), the presence of a trained, external facilitator was the 
factor students in this study most strongly supported; a factor also identified as critical in the work of Clark 
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and Bekey (1979). Having the opportunity to express ideas; anonymity; satisfaction in knowing the 
results would be listened to by the stakeholders in the program; and having clear guidance for the process 
were themes that appeared in comments in both this study and in Clark and Bekey's work. In regard to the 
closely related topic of interaction between learner and instructor, Dillon, Gunawardena, & Parker (1992) 
found that such interactions, during class time, were viewed as the most important form of interaction with 
the distance education students they studied. 

Although Clark and Bekey's procedure for administering the SGID requires that the facilitator's 
visit not be announced in advance to avoid student absences by some who might feel strongly about the 
class, unlike Clark and Bekey's process, SGID at the University of Hawaii, provides prior notice. This 
study followed that pattern, with no change noted in the attendance patterns. The participants in this study 
provided further support for the idea of prior notice, by recommending that the three questions be 
distributed prior to the process. 

Students offered few comments on the audio and video delivery system, despite Walker & 
Hackman's finding (1992) that clear audio and video were significant factors for distance education students. 
Given the ease and technical sophistication with which the HITS delivery system functions at the 
University of Hawaii, this finding was not unexpected for these students. 

Following guidelines for using evaluations for improving instruction (Maitino, 1992; Theall & 
Franklin, 1990), the direct effects of the SGID results were the inter and intra university and school 
departmental meetings to hear the results of the SGID and to plan for recommended changes. While many 
factors figured in the advancement of tentative plans for developing a technology specialist certificate within 
the Department of Education and the concomitant graduate level training, an examination of the timeline in 
Figure 2 shows the cloce proximity of the major thrust of that work to the SGID. The university and 
school stakeholders focused formally on a Task Force charged with identifying needs and defining the role 
for such a professional after receiving the results of the evaluation process. 



Implications 

While some studies show linkages of student achievement to students' ratings of courses (Cohen, 
1981), others (Byrne, 1992) question the validity of using such data, because of the dangers inherent in 
obtaining reliable estimates of coefficients from small samples, particularly if employment decisions for 
instructors are based on such results. Given the small number of participants and lack of legal access to 
student grades, this study made no attempt to examine the relationship between the students' ratings and 
their academic achievement in the courses. Future researchers with larger samples may wish examine such 
quantitative data in relation to the qualitative data the SGID provides. The results of this study should be 
interpreted with caution as a descriptive study of informal linkages. 

Others may wish to draw upon the ease with which the methods employed in this study obtained 
cooperation from the participants and with the ability to gather a wide range of perspectives. 

The pattern that emerged from this research is the critical role the facilitator plays in SGID. 
Future studies need to further investigate the significance of this effect. 

In light of the recognized importance of perceived interactivity to distance education student 
satisfaction (Fulford & Zhang, 1994), future studies may also wish to investigate the relation of SGID in 
contributing to student perceptions of interactivity within their distance education environment. 

The SGID provides a structure that meets guidelines for obtaining student ratings for 
improvement. Seldin (1988) has defined these guidelines as: asking diagnostic questions; having open- 
ended response format support feedback on an infinite variety of factors; maintaining confidentiality for 
instructors; providing questions that students are capable of answering; encouraging instructors to actively 
participate by adding any specific questions they may have; being able to use other evaluation instruments; 
supporting comprehensive student involvement; and providing written comments. SGID accommodates all 
these guidelines. 

The assessment of student attitudes toward their distance education milieu is one of the factors that 
continues to appear in the literature on distance education (Thorpe, 1988; Wagner, 1993). This study 
attempted to link research and evaluation of distance education in the area of student learning in relation to 
graduate level in-service teachers located on four islands in Hawaii. 

Given the factor that college courses delivered by television reach more adult students at more colleges than 
any other type of technology-based course (Brock 1991), there is a need to continue investigating evaluation 
methods for such distance education courses. To fill the continuing need for rigorous practical research on 
assessment for distance education (Holmberg, 1991), future studies with diverse distance education 
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populations could be designed to employ Small Group Instructional Diagnosis, which, with its potential 
for gaining broad insights into students perceptions and attitudes about distance education courses, in 
combination with quantitative measures, has the potential for providing additional insights into student 
satisfaction in distance education settings. 



References 

Birenbaum, H., Glide, J., & Kinsler, K. (1994). Trends and developments in broad and two-way video 

distance education. Ed Journal . 7 (10), J1-J7. 
Brock, D. (199l).[Summary]. Symposium on telecomm unications and the adult leader - Washington, DC: 

American Association of Community and Junior Colleges. 
Byrne, C. J. (1992). Validity studies of teacher rating instruments: Design and interpretation. Research in 

Clark, D. J. & Bekey, J. (1979). Use of small groups in instructional evaluation. Insight to Teaching 
Excellence. 1 m. 2-5. 

Cohen, P A. (1981). Student ratings of instruction and student achievement: A meta-analysis of 

multisection validity studies. Review of Educational Research. 51 (3), 281-309. 
Dillon, C. L., Gunawardena, C. N., & Parker, R. (1992). Learner support: The critical link in distance 

education. Distance Education. 13 (1), 29-45. 
Egan, M. W., Welch, M., Page, B., Sebastian, J. (1992). Learners' perceptions of instructional delivery 

systems: Conventional and television. The American Journal of Distance Education. 6 (2). 47-55. 
Fulford, C. & Zhang, S. (1994). Predicting student satisfaction from perceptions of interaction in distance 

learning. In G. Davies & B. Samways (Eds.). Proceedings of the IFIP TC3 Third Teleteaching 

Conference (pp. 259-268). Amsterdam: Elsevier Science Publishers. Teleteaching. 
Holmberg, B. (1991). Distance education as communication: The impact of research and the requirements of 

practitioners,. In Author (Ed.) Research into distance education . New York: Peter Lang. 
Holmberg, R. G. & Bakshi, T. (1992). Postmortem on a distance education course: Successes and failures. 

The American Journal of Distance Education. 6 (1), 27-39. 
Kember, D., Lai, T., Murphy, D., Siaw, I., & Yuen, K.S. (1992). A synthesis of evaluations of distance 

education courses. British Journal of Educational Technology . 23 (2), 122-135. 
Lauzon, A. C. (1992). Integrating computer-based instruction with computer conferencing: An evaluation 

of a model for designing online education. The American Journal of Distance Education . fi (2), 32- 

46). 

Maitino, J. R. (1992). Using student evaluations and other measures of instruction to improve teaching. 

Journal of Staff. Program and Organizational 

Development. 1Q f4V 209-215. 
Moore, M. G. (1989). Editorial: Three types of interaction. The American Journal of Distance Education. 3 

(2), 1-6. 

Patton, M. Q. (1987). How to use qualitative methods in evaluation . Beverly Hills, CA: Sage. 
Reinharz, S. (1993). Empty explanations for empty wombs: An illustration of secondary analysis of 

qualitative data. In M. Schratz fEd.V Qualitative voices in educational research (pp. 157-178). 

Washington, DC: Falmer Press. 
Richardson, V. Conducting research on practice. Educational Researcher. 23 (5). 

5-9. 

Schlosser, C. A. & Anderson, M. A. (1994). Distance education: Review of the literature . Washington, 

DC: Association for Educational Communications and Technology. 
Seldin, P. (1988). Evaluating college teaching. In R. E. Young & K. E. Eble (Eds.), College teaching and 

learning: Preparing for new commitments: No. 33. 

New directions for teaching and learning. San Francisco: Jossev Bass . 
Theall, M. & Franklin, J. (Eds.). (1990). Student ratings of instruction: Issues for improving practice: No. 

43. New di rections for teaching and learning . San Francisco: Jossey Bass. 
Thorpe, M. (1988). Evaluating open and distance learning. (2nd ed.) Essex, UK: Longman Group. 
Verduin, J. R., Jr. & Clark, T. A. (1991). Distance education: The foundation of effective practice. San 

Francisco: Jossey Bass. 



107 



Wagner, E. (1993). Variables affecting distance educational program success. Educational Technology. 33 
(4), 28-32. 

Walker, K. & Hackman, M. (1992) Multiple predictors of perceived learning and satisfaction: The 

importance of information transfer and non-verbal immediacy in the televised course. Distance 

Educate 

Willey, G. (1991). Telecommunications and the adult learner. In D. Brock, [Summary]. Symposium QH 

telecommunications ant] the gdult learne r. Washington, DC: American Association of Community 
and Junior Colleges. 

Willis, B. (1991). Evaluation. In M. McGill (Ed.), A faculty gu ide to distance education. Boulder. CO: 

Western Interstate Commission for Higher Education 
Zvacek, S. M. (1994). Evaluating distance education programs using a systems approach. Paper presented at 

the meeting of the Association for Educational Communication Technology, Nashville, TN. 



Author Note 

Annette C. Sherry, Department of Educational Technology, College of Education, University of 
Hawaii at Manoa; William F. Burke, Center for Teaching Excellence, University of Hawaii at Manoa (now 
at the Biology Program, University of Hawaii at Manoa). 

This study was conducted to obtain information on the effect of the Small Group Instructional 
Diagnosis for use not only within the college and within the university's Center for Teaching Excellence, 
but to encourage others to investigate its effectiveness in studies on improving learning. 

Bill Burke and I would like to thank the members of the T 3 cohort for their time and thoughtful 
responses as they participated in this study. 

Correspondence concerning this article should addressed to the principal investigator, Annette C. 
Sheny , Department of Educational Technology, College of Education, 1776 University Avenue, Wist 23 1 , 
Honolulu, Hawaii 96822. Electronic mail may be sent via Internet to asherry@uhunix.uhcc.Hawaii.Edu. 
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Table 1 

Typology of Student Reactions to Small Group Instructional Di agnosis Model bv Sites 

SilfiS 



Types of Interaction in Evaluation 

Process All Campus Remote 



Helpful Interactions 














Learner-Process 


44 


(41.0) 


23 


(44.2) 


21 


(37.5) 


Learner-Facilitator 


44 


(41.0) 


19 


(36.5) 


25 


(44.6) 


Learner-Learner 


20 


(18.5) 


10 


(19.2) 


10 


(17.2) 


Hindering Interactions 














Learner-Process 


28 


(62.2) 


11 


(50.0) 


17 


(74.0) 


Learner-Facilitator 


1 


(2.2) 


0 


(0.0) 


1 


(4.3) 


Learner-Learner 


8 


(17.7) 


5 


(22.7) 


3 


(13.0) 


None 


8 


(17.7) 


6 


(27.3) 


2 


(8.7) 


Change Interactions 














Learner-xYocess 


46 


(85.2) 


24 


(77.4) 


22 


(95.7) 


Learner-Facilitator 


0 


(0.0) 


0 


(0.0) 


0 


(0.0) 


Learner-Learner 


0 


(0.0) 


0 


(0.0) 


0 


(0.0) 


No Changes 


8 


(14.8) 


7 


(22.6) 


1 


(4.3) 



Note, n is based on number of items, not respondents, n = 207 for all items. 

H = 105 for on-campus items, n = 102 for remote campus items. Percentages figured on 100% for the 
three main types of interaction appear in parentheses. 
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Table 2 

Ranking of Student Comments Concernin g Distance Education Evaluation Model Related to Types of 
Interaction bv Site: Hel pful Interactions 

Rank by Sites 

Types of Interaction in Evaluation Process All Campus Remote 

Helpful Interactions 



Learner-Process 

Structured questions 

Opportunity to express ideas 

A third party guide 

Non-threatening environment 

Meaningful dialogue focused on improvement 



Learner-Facilitator 

Overall Skills: professional/excellent/leadership 1* l a l a 

Atmosphere Creation Ability: supportive of 2 ia 

pardcipadoii/neutral/comformble/opportunities to respond 

Facilitation Skills: explicit in instructions/everyone kept on track - 2 

Mediation Skills: objective; effective mediator/ honest/non- - 2 

judgmental 



Learner-Learner 

Hearing ideas of others/group memory/collective 1 2 
voices/unanimity/learning from each other 

Professionalism of groups 2 i a 



a Most frequently mentioned item within all types of Helpful Interaction categories. Rank based on 
frequency count of items for All Sites, a = 207; for On-Campus Sites, a = 105; for Remote Sites, n = 
102. 



Table 3 

Ranking of Student Comments Concerning Distance Educati on Evaluation Model Related to Types of 
Interaction b y Site: Hindering Interactions 

Sites 

Typec ox Interaction in Evaluation Process All Campus Remote 



Hindering Interactions 
Learner-Process 



Amount of time for process 


l a 


l a 


l a 


Time Limits 


2 


1 


1 


Personal feelings 


2 




2 


Learner-Facilitator 








Facilitator monitor time 


1 




1 


Learner-Learner 








Peer pressure 


1 


1 


1 


Feelings about self 


2 




2 


No Problems 








Not difficult 


1 


1 


1 



a Most frequently mentioned item across all types of Hindering Interactions 
categories. Rank based on frequency count of items for All Sites, n = 207; for On- 
Campus Sites, n = 105; for Remote Sites, n = 102. 
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Table 4 



Ranking of Student Comment s Concerning Distance Education Evaluation Model Related to Types nf 
Interaction: Change Interactions 

Sites 

Types of Interaction in Evaluation Process 
Changes Interactions 
Learner-Process 

Conduct earlier in the semester 

Provide the questions prior to the session 
No Changes 

Continue/don't change 

Would like to use format 



a Most frequently mentioned item across all types of Change Interactions categories. Rank based on 
frequency count of items for All Sites, a = 207; for On-Campus Sites, a = 105; for Remote Sites, n = 
102. 



AH Campus Remote 



l a ia ia 

2 2 2 

1 t a 

2 2- 



Figure L. Percentage Distribution by Sites for Items on Effectiveness of Small Group Instructional 
Diagnosis 




0 Help Items 
M Hinderance Items 

1 No Hindrance Items 
(HI Change Items 
I No Change Items 
□ Not Applicable Item 
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Sites 



Figure 2, Timeline of Activities in Relation to Small Group Instructional Diagnosis 
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